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During the 2013-2014 academic year, 647 elementary, middle, and high-school students participated in 
the New Digital Energy Game (NDE) game in the Houston Independent School District (HISD). This is the 
fourth year that NDE has been implemented in HISD. The game was designed to enrich students’ science 
experiences in order to stimulate their interests to pursue science energy careers. The game was funded by 
Chevron Corporation and developed by Tietronix, Inc. Paired t-test analysis yielded a sample of 382 
elementary (28%), middle (61%), and high (11%) school students. Based on a 4-point Likert-type scale, the 
majority of students at each level indicated either no change or a slight to moderate positive change in 
their science interests and attitudes from pre- to post-tests. The majority of elementary, middle, and high- 
school students showed a loss in their science knowledge assessment scores, comparing before and after 
game participation results. There was a positive relationship between post-interests in science and post- 
attitudes about science following game participation. Propensity score, nearest neighbor matching yielded 
slightly lower 2014 Stanford 10 science performance of NDE students compared to a similar student group, 
suggesting no positive benefit of the game using this measure. 


Background 


Much attention has been devoted toward 
understanding computer games and _ their 
educational benefits to students. Computer games 
that provide instruction have the potential to 
stimulate the academic environment and increase 
students’ awareness and knowledge of 
phenomenon to solve problems (Honey & Hilton, 
2010; Jones, 1996; Mundie, 2011; Owston, 2009; 
Yang, Kun, & Chein, 2010). “Electronic gaming 
may be one way to engage students in critical 
thinking necessary to apply subject matter to "real- 
life" experiences” (Curriculum Review, p. 10). 
Digital games may encourage students to ask 
unanswered questions and teach them to use 
problem-solving techniques. 

Brendzel (2004) pointed out that “games provide 
a natural motivation” (p. 32) to apply good 
teaching strategies to help build science concepts. 
Contest can further stimulate students’ drive to 
compete, which may improve comprehension in a 
format that may be interesting to students. 
Interested students are more engaged in activities 


that they value (Bulunuz & Jarret, 2009). Ogunkola 
(2011) found that increased attitudes and interest in 
an activity is correlated with good study habits and 
engagement in school. Disengaged students are 
more likely to have poor attendance and more 
likely to drop out of school (Balfanz, Herzog, & 
Mac Iver, 2006). 

The New Digital Energy (NDE) game focused 
on increasing students’ understanding of science 
energy concepts. Game tasks accommodated all 
student academic levels by combining strategy, 
construction, and game management, and requiring 
that players build energy companies and gain 
dominant market share to meet the needs of U.S. 
cities. Students played as teams against artificial 
intelligence, competing across three levels of 
difficulty. Variations in difficulty were 
incorporated into lessons that students had to 
master to open options within the game. Lessons 
and questions prompted students to game-play 
decisions that required understanding of physics, 
chemistry, earth science, and math concepts. 

To that end, this study measured the effects of 
the NDE game on students’ acquisition of science 
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Figure 1. Number of students who registered and played the NDE game, 2010-2011 through 2013-2014 


knowledge and science achievement. In addition, 
the study assessed the effects of the game toward 
increasing students’ interests and attitudes in 
science. 


Methods 
All Study Participants 

All HISD elementary, middle, and high-school 
science teachers were invited to recruit students to 
participate in the NDE game during the fall of 
2014. Teachers were required to register and sign a 
letter of commitment to participate. Figure 1 
presents the number of students who registered 
along with the number of students who played the 
game over the past four academic years. Twenty- 
two schools and 29 teachers were represented in 
the 2012-2013 data; whereas, seven elementary 
and nine secondary schools (eight middle schools 
and one high school) were represented in the 2013— 
2014 data. Students playing the game dropped by 
20.6 percent from the 2012-2013 to the 2013-2014 
academic year. There was also a drop in 
registration (-16.3%) and participation (-67.6%), 
from the previous to the current academic year. 


Student Sample 

The study sample consisted of 107 elementary, 
233 secondary, and 41 high school students who 
completed the four instruments (pre- and post- 
science energy knowledge assessments along with 
pre- and post-science interest/attitude surveys). 
Students in the matched sample were enrolled at 
seven elementary schools, eight middle schools, 
and one high school. Table 1 presents 
demographic characteristics of the matched student 
sample. The majority of students were male 
(between 52.8% and 76.0%) and economically 


disadvantaged (between 78.5% and 87.0%). The 
highest proportion of at-risk students was at the 
elementary level (58.3%) compared to the lowest 
percentage at the middle-school level (23.6%) 
There were nearly twice as many limited English 
proficient (LEP) students at the elementary level 
than at the middle-school level (34.3% vs. 17.2%). 
High-school students were only 4.9% LEP. 
Comparatively, the District had a higher percentage 
of at-risk students (68.7%) and a lower percentage 
of G/T students (15.6%) than the NDE student 
sample at all levels (PEIMS, 2013-2014). 


Measures and Variables 

Students were administered four web-based 
surveys accessed at the game site. The instruments 
included a 10-item science interest survey, a 10- 
item science attitudes survey, and a 17-item 
multiple-choice science energy assessment using a 
pre- and post-test design. The survey items 
measuring interests and attitudes had good internal 
consistency, with a Cronbach alpha coefficient of 
.81 and .86, respectively. While students were 
encouraged to complete the instruments, no 
incentives were offered for completion. 


Table 1: Profile of NDE Student Sample by School Level, 
2014 


Elem Middle High HISD 
(n = 108) (n = 233) (n = 41) 

Gender % % % % 
Male 52.8 76.0 56.1 51.0 
Female 47.2 24.0 43.9 49.0 

Eco. 87.0 78.5 82.1 80.4 

Disadv. 

At Risk 58.3 23.6 43.9 68.7 

G/T 21.3 33.9 31.7 15.6 

LEP 34.3 17.2 49 29.5 

Spec. Ed. 8.3 LEZ. 0.0 7.7 
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Elementary-School Student Sample 


Had fun when learning science 

Talked to my teacher about science 

Participated in science competitions 

Talked to my friends about science 

Attended science group 

Visited websites about science 

Participated in after-school science activities 
Talked about science to parents or family members 


Watched news on TV that involved science concepts 


Read books or magazines about science outside of school 
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Figure 2: Percentage of elementary-school students whose interest in science ratings before and after participation in NDE showed 
a positive change (1 to 3 points), no change (0 points), or a negative change (-1 to -3 points) based on 4-point Likert-type scale 


Interests were rated using a Likert-type scale: very 
often-4; regularly-3; sometimes- 2; and never or 
hardly ever- 1. Attitudes were rated strongly agree- 
4; agree-3, disagree-2, and strongly disagree-1. 

The percentage of students whose ratings 
increased by one to three points, did not change, or 
decreased by one to three points from pre- to post- 
test was analyzed using IBM SPSS. Comments 
about students’ experiences were summarized. 
Descriptive statistics and correlation analysis was 
conducted between the measures. Propensity score 
matching was used to estimate the probability of 
being exposed to treatment given a set of observed 
variables. Propensity score matching is a common 
practice in social science to make causal inferences 
based on observational data (Cohen, 1988). NDE 
students who were included in the “study sample” 


were used in the propensity score matching. 
Economic status, gender, grade level, special 
education program status, and current 2014 
Stanford reading normal curve equivalent scores 
were the control variables. 


What was the impact of the NDE game on 


students’ science interests and attitudes? 


Figure 2 and Table 2 depict interest and attitude 
survey results of the elementary student sample 
before and after participation in the NDE game in 
2014. The interest items that showed the highest 
percentage of “positive change” were “visited 
websites about science” and “watched news on TV 
that involved science concepts” (33.3% and 32.4%, 
respectively). 


Table 2. Science Attitudes of Elementary-School Student Sample Before and After Participation in the NDE game, 2014 


Positive No Negative 
Change Change Change 
(+) ian 
% % % 
I am interested in learning more science. 20.6 50.5 29.0 
Science will help me become successful. 17.1 60.0 22.9 
I like studying science. 22.8 56.4 20.8 
A science energy game will help me learn difficult science theories that I may not 
‘ nace 22.4 46.7 30.8 
understand without seeing it in the game. 
Studying science is worthwhile because it will improve my career opportunities. 23.1 48.1 28.7 
Making an effort to study science is worth it because it will help me get into college. 17.8 55.1 AGfedl 
I study science because I know it is useful to me. 11.1 58.3 30.6 
I would like to have a career involving science. 21.9 46.7 31.4 


@ Negative Change (-) 


@ Positive Change (+) 
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Middle-School Student Sample 


Had fun when learning science 

Talked to my teacher about science 

Participated in science competitions 

Talked to my friends about science 

Attended science group 

Visited websites about science 
Participated in after-school science.. 
Talked about science to parents or.. 
Watched news on TV that involved.. 


Read books or magazines about.. 


= Negative Change (-) 
@ No Change 


@ Positive Change ( +) 
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Figure 3: Percentage of middle-school students whose interest in science ratings before and after participation in NDE showed 
a positive change (1 to 3 points), no change (0 points), or a negative change (-1 to -3 points) based on 4-point Likert-type scale 


“No change” was most likely on the item “had fun 
when learning science” (62.1%). The highest 
percentage of “negative change” in ratings was on 
the item “talked to my friends about science”. 
Regarding their attitudes about science, the item 
that reflected the highest percentage of “positive 
change” was “study science is worthwhile because 
it will improve my career opportunities” (23.1%). 
“Science will help me be more successful” showed 
the highest percentage of “no change” (60.0%); 
whereas, the item that showed the highest 
percentage of “negative change” was that “I would 
like to have a career involving science” (31.4%). 
Science interest and attitude results of middle- 


school students before and after participation in the 
game can be found in Figure 3 and Table 3. The 
science interest item that reflected the highest 
percentage of “positive change” from pre- to post- 
test was “reading books or magazines about 
science outside of school” (30.0%) and 
“participating in afterschool science activities” 
(28.3%) (Figure 3). 

Relative to attitudes (Table 3), the middle-school 
students were more likely to express “positive 
change” on the item “I like studying science” 
(20.5%). At the same time, the highest percentage 
of “no change” was on “making an effort to study 
science is worth it because it will help me get into 
college” (82.9%). 


Table 3. Science Attitudes of Middle-School Student Sample Before and After Participation in the NDE game, 2014 


Positive No Negative 
Change Change Change 
(+) (+) 
% % % 
I am interested in learning more science. 14.6 48.8 36.6 
Science will help me become successful. 14.6 61.0 24.4 
I like studying science. 20.5 51.3 28.2 
A science energy game will help me learn difficult science theories that I may not 
: Se cuand 14.6 43.9 41.5 
understand without seeing it in the game. 
Studying science is worthwhile because it will improve my career opportunities. 10.3 71.8 79) 
Making an effort to study science is worth it because it will help me get into college. 17.1 82.9 31.7 
I study science because I know it is useful to me. 12.2 48.8 39.0 
I would like to have a career involving science. 10.0 57.5 32.5 
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High -School Student Sample 


Had fun when learning science 

Talked to my teacher about science 

Participated in science competitions 

Talked to my friends about science 

Attended science group 

Visited websites about science 

Participated in after-school science activities 
Talked about science to parents or family.. 
Watched news on TV that involved science. . 


Read books or magazines about science.. 


& Negative Change (-) 
= No Change 


@ Positive Change ( +) 
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Figure 4: Percentage of high-school students whose interest in science ratings before and after participation in NDE showed a 
positive change (1 to 3 points), no change (0 points), or a negative change (-1 to -3 points) based on 4-point Likert-type scale 


“Negative change” was most evident on “a science 
energy game will help me learn difficult science 
theories that I may not understand without seeing it 
in the game” (41.5%). 

The results of the high-school student sample 
regarding their interests and attitudes about science 
are shown in Figure 4 and Table 4, respectively. 
The science interest item that showed the highest 
percentage of “positive change” was “participated 
in science competitions” and “talked to my friends 
about science” (43.9%). “No change” was most 
evident on the item “watched news on TV that 
involved science concepts” (75.6%). “Negative 
change” was also more likely on this item (29.3%) 
(Figure 4). 


“Positive change” in attitudes about science for 
high-school students was highest on the item “I 
study science because I know it is useful to me” 
(27.1%) (Table 4). The highest percentage of 
students expressed “no change” on the item that 
“making an effort to study science is worth it 
because it will help me get into college (59.4%). At 
the same time, the highest percentage of “negative 
change” was reflected on “a science energy game 
will help me learn difficult science theories that I 
may not understand without seeing it in the game” 
(32.6%). 


Table 4. Science Attitudes of High-School Student Sample Before and After Participation in the NDE game, 2014 


Positive No Negative 
Change Change Change 
(+) (-) 
% % % 
I am interested in learning more science. 22.3 48.1 29.6 
Science will help me become successful. 24.7 53.7 21.6 
I like studying science. 24.2 54.3 21.5 
A science energy game will help me learn difficult science theories that I may not 
: aa 2800 43.9 32.6 
understand without seeing it in the game. 
Studying science is worthwhile because it will improve my career opportunities. 22.0 50.4 27.6 
Making an effort to study science is worth it because it will help me get into 16.6 59.4 24.0 
college. 
I study science because I know it is useful to me. bfell 46.3 26.6 
I would like to have a career involving science. 20.6 53.9 25.4 
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Table 5. Science Knowledge Assessment Raw Score Results of the Student Sample Before and After Participation in the NDE game, 


2014 Se 
Gain Loss No 
(+) (-) Diff. 
Raw Score Raw Score 
n % Min Max Mean n % Min Max Mean n % 
Elementary (n=108) 31 28.7 5.9 29.4 10.6 66 61.1 5.9 47.1 16.4 11 10.2 
Middle (n=233) 89 38.2 5.9 41.2 13.8 100 42.9 Se) 52.9 13.2 44 18.9 
High (n=41) 11 26.8 5.9 17.7 10.7 26 63.4 5.9 52.9 17.9 4 9.8 
Total (n=382) 131 343 5.9 41.2 12.8 192 50.3 be) 52.9 15.0 59 15.4 
What was the effect of the NDE game on students overall showed any gain in_ their 


students’ science knowledge? 


Table 5 shows the mean “gain”, “loss”, and “no 
difference” in the science knowledge raw scores 
from pre- to post-assessment of the student sample. 
The minimum and maximum number of points that 
students gained or lost along with the number and 
percentage of students who had gains, losses, and 
showed no difference in their science knowledge 
assessment raw scores from pre- to post- 
assessment are presented. 

Middle school students achieved the highest 
mean “gain” in their raw score on the knowledge 
assessment (13.8 points). Elementary and high- 
school students attained comparable gains in their 
raw scores (10.6 and 10.7 points, respectively). 
High-school students attained the highest mean 
“loss” in their raw score on the assessment over 
time. The highest percentage of the middle-school 
student sample had “no difference” in their science 
assessment raw score before and after participation 
in the game. Over two-thirds of the high and 
elementary-school students showed either a loss or 
no change on the assessment. Only one-third of the 


knowledge assessment raw score. 


What were the associations between science 


interests, attitudes, and science assessment 
results before and after participation in the 
NDE game? 


Table 6 depicts means and standard deviations 
along with intercorrelations of the measures for the 
elementary-school student sample. There were 
slight decreases in their mean interest (-.08) and 
attitude ratings (-.13) as well as a moderate 
decrease in their mean science knowledge 
assessment scores (-6.97 points) from pre- to 
posttest. Other notable findings included a positive 
association between elementary-school students’ 
post-interests, post-attitudes, and post-science 
knowledge assessment results. In other words, as 
elementary students’ interests in science increased, 
their attitudes and science knowledge assessment 
scores increased after participation in the game. 
Pre- and post-attitudes as well as pre-knowledge 
intercorrelations with post-knowledge were highly 
statistically significant at p<.01. 


Table 6, Summary of Intercorrelations, Means, and Standard Deviations for Scores on the Pre- and Post-Science Interest, Attitude, 


and Knowledge Measures for the Elementary Student Sample, 2014 


1 2 3 4 5 6 
1. Pre-Interest - .622™ 560" .390™ -.048 .006 
2. Post-Interest .622™ - 520" 656" 100 120 
3. Pre-Attitude 560" 520" : 628" 113 292" 
4, Post-Attitude 390" 656" 628" - .176 311" 
5. Pre-Knowledge -.048 .100 113 .176 2 625" 
6. Post-Knowledge 006 120 .292" 311" 6257 ; 
M 2.66 2.58 3.10 2.97 45.10 38.13 
SD 58 65 63 68 15.98 18.11 


**D<.01 
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Table 7. Summary of Intercorrelations, Means, and Standard Deviations for Scores on the Pre- and Post-Science Interest, Attitude, 
and Knowledge Measures for the Middle-School Student Sample, 2014 


1 2 


1. Pre-Interest - 595" 

2. Post-Interest 595" - 

3. Pre-Attitude 525" 456" 

4. Post-Attitude 403” 629" 

5. Pre-Knowledge .030 -.011 

6. Post-Knowledge .020 .042 
M 2.37 2.41 
SD 56 .59 


**p < 01 


3 4 5 6 
525" 403" .030 020 
456" 629” -.011 042 
- 568" 072 037 
568" : .082 110 
072 .082 : 690" 
037 .110 690" - 
3.00 2.90 56.22 55.79 
67 77 17.10 18.95 


Table 8. Summary of Intercorrelations, Means, and Standard Deviations for Scores on the Pre- and Post-Science Interest, Attitude, 
and Knowledge Measures for the High-School Student Sample, 2014 


1 2 

1. Pre-Interest - .409™ 

2. Post-Interest 409" - 

3. Pre-Attitude 371" 309" 
4, Post-Attitude .232 314" 
5. Pre-Knowledge .299 113 
6. Post-Knowledge 235 -.154 
M 2.63 2.76 
SD 64 .63 


*p < .05; **p < .01 


Middle-school students’ results can be found in 
Table 7. There was a slight increase in their mean 
interests rating (+.04) from pre- to post-survey. 
However, there were slight decreases in their mean 
attitudes rating (-.10) and mean science knowledge 
assessment score over the same time period (-.43 
points). Regarding the post-survey measures, as 
post-interests in science increased (p < .01), post- 
attitudes (p < .01), and post-science knowledge 
increased. 

Data on high-school students is shown in Table 
8. The mean interests rating increased slightly 
(+.13) from pre- to post-test, while the mean 
attitudes rating decreased (-.29) over the same time 
period. There was a moderate decrease in students’ 
mean science knowledge assessment score (-8.47) 
before and after participation in the game. As post- 
interests in science increased (p <. .05), post- 
attitudes increased (p < .05), and post-science 
knowledge decreased. 


3 4 5 6 
371" 232 299 235 
309° 314° 113 -.154 
- 501” .074 063 
501” - .099 210 
074 .099 “ 677" 
063 .210 677" = 
3.11 2.82 65.14 56.67 
63 67 18.62 23.00 


What was the effect of the NDE game on 
science achievement? 


Propensity score, nearest neighbor matching was 
used to determine the effect of the NDE program 
on students’ 2014 Stanford science NCE scores 
(See Appendix A). The statistical model used in 
the analysis controlled for economic status, grade 
level, gender, special education program status, and 
2014 Stanford 10 reading NCE scores. Bias 
analysis was conducted and yielded a comparison 
sample where the treated and untreated shared the 
same characteristics, which means that the 
selection bias had been mitigated in the “new” 
sample (Austin, Grootendorst, and Anderson, 
2007). Out of the 272 NDE students in the initial 
sample, Stata selected 270 students for the model. 
Out of the 53,627 students in the comparison 
group, Stata selected 52,404 students for the 
model. Prior to controlling for these variables, the 


NDE students had a higher science score than the 
control group (58.1 vs. 52.6 NCEs). Propensity 
score matching yielded slightly lower science 
performance due to participation in the program 
compared to students who did not participate in the 
program (-1.3 NCEs). Although the difference was 
not statistically significant, the model suggested 
that the NDE program did not have a positive 
effect on students’ Stanford science NCE scores. 


Discussion 


The New Digital Energy game was designed to 
increase student’s understanding of science energy 
concepts by introducing game construction and 
management strategies. Students worked as teams 
to build energy companies, gain dominant market 
share, and meet the needs of cities throughout the 
United States. Students played against artificial 
intelligence to answer challenging questions about 
science energy. An objective of the game was to 
increase students’ interests and attitudes about 
science as they increased their knowledge of 
science energy concepts. 

There were several limitations to the study. First, 
given the team format, the study lacked data on 
actual time spent by individual students playing the 
game. This could have affected students’ 
experiences gaining science energy knowledge and 
influenced their interests and attitudes about 
science at post-test. Another limitation was the lack 
of a comparison group to assess the impact of the 
game on students’ interests and attitudes. While a 
pre- post-test design provided an_ alternative 
method for comparing students, threats to validity 
could still exist relative to knowing whether other 
external factors influenced the changes in student 
outcomes over time (Boyd, 2002). In consideration 
of these threats, propensity score matching was 
used to reduce bias and assess whether 
participation in the game had an effect on students’ 
science achievement while controlling for key 
demographic characteristics. No benefit was 
observed in student’s science performance on the 
2014 Stanford 10 achievement test. Finally, the 
lack of data at the four data collection points on all 
students who played the game resulted in 
substantial attrition. (The number of students who 
played the game was substantially lower than the 
number of students who initially registered (647 
out of 2851 students). Even less students were 
included in the final analysis of the data (382). The 
recruitment process relied on teacher commitment, 
which may have contributed to attrition of the 
initial group of student participants. 
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The majority of elementary, middle, and high- 
school students showed either no change or a slight 
to moderate positive change in their science 
interests and attitudes from pre- to post-tests. 
Moreover, the majority of students at all levels had 
a loss in their science knowledge assessment scores 
over the same time period. Among the most 
notable positive findings were as students’ interests 
in science increased after participation in the game, 
their attitudes about science also increased after the 
game. Also, there was a positive association 
between elementary and middle-school students’ 
post-interests in science and _post-science 
knowledge. However, for high-school students, as 
post-interests increased, post-science knowledge 
decreased. This finding seems inconsistent with the 
expectation that a positive relationship should exist 
between interests and achievement (Brendzel, 
2004). Moreover, previous year results were 
consistently positive (Holmes, 2013). While each 
year, a different student sample may have 
participated in the program, a recommendation is 
to further explore students’ perceptions about the 
game in more detail. This could be done through 
focus groups or one-on-one interviews with 
students who have participated in the game over 
the past four years. Information gathered from 
students could be incorporated in future game 
development. Another recommendation is to 
reconsider strategies on successful recruitment of 
science teachers and students from more HISD 
schools. Finally, while evidence of improvements 
in students’ science knowledge was not evident in 
this study, what may be more important is that 
students were exposed to science material needed 
to enhance their understanding of science concepts 
(Wainwright & Linebarger, 2006). These 
experiences may be valuable to students who, 
otherwise, could not afford them. 
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Appendix A 


Propensity Score Matching Results 


New Digital Energy Game, 2013-2014 


Treated Controls 
(n = 270) (52,404) 
Mean Mean Difference S.E. T-stat 
Science NCE Science NCE 
Unmatched 58.1 52.6 55.6 13.93 3.99 
Matched 58.1 59.4 -13.1 18.11 -.072 
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